chore(config): bump ACM/MCE operator bundles (bisect e2e #5789)#5911
Merged
Conversation
|
Skipping CI for Draft Pull Request. |
Collaborator
Author
|
/hold |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR bumps the ACM operator bundle and MCE operator bundle digests (and the derived ACM/MCE Helm chart content + fixtures) to isolate whether that specific bump is responsible for the e2e-parallel CreateHCPCluster timeout regression being bisected from PR #5789.
Changes:
- Update default ACM operator bundle digest and MCE operator bundle digest in
config/config.yaml. - Propagate the updated digests into rendered environment configs under
config/rendered/dev/.... - Update ACM/MCE Helm charts and the Helm template golden fixture to reflect the bumped bundle/app versions and operand image digests.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| config/config.yaml | Bumps the default ACM and MCE bundle digests (and version comments) used for materialization. |
| config/rendered/dev/pers/westus3.yaml | Updates rendered ACM/MCE bundle digests for this environment. |
| config/rendered/dev/perf/westus3.yaml | Updates rendered ACM/MCE bundle digests for this environment. |
| config/rendered/dev/dev/westus3.yaml | Updates rendered ACM/MCE bundle digests for this environment. |
| config/rendered/dev/cspr/westus3.yaml | Updates rendered ACM/MCE bundle digests for this environment. |
| config/rendered/dev/ci00/centralus.yaml | Updates rendered ACM/MCE bundle digests for this environment. |
| config/rendered/dev/ci01/centralus.yaml | Updates rendered ACM/MCE bundle digests for this environment. |
| acm/deploy/helm/multicluster-engine/Chart.yaml | Updates MCE chart appVersion/version and sources digest to match the bumped MCE bundle. |
| acm/deploy/helm/multicluster-engine-crds/Chart.yaml | Updates MCE CRDs chart appVersion/version and sources digest to match the bumped MCE bundle. |
| acm/deploy/helm/multicluster-engine/templates/multicluster-engine-operator.deployment.yaml | Updates operator/operand image digests and operator version env var consistent with the new MCE bundle. |
| acm/deploy/helm/multicluster-engine-config/charts/policy/Chart.yaml | Updates policy chart version pins to the newer ACM version. |
| acm/deploy/helm/multicluster-engine-config/charts/policy/charts/grc/Chart.yaml | Updates grc chart app/version to the newer ACM version. |
| acm/deploy/helm/multicluster-engine-config/charts/policy/charts/cluster-lifecycle/Chart.yaml | Updates cluster-lifecycle chart app/version to the newer ACM version. |
| acm/deploy/helm/multicluster-engine-config/charts/policy/values.yaml | Updates ACM governance-related image override digests used by the policy chart. |
| acm/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_mce.yaml | Updates the Helm template golden fixture to reflect the updated MCE operator version. |
Collaborator
Author
|
/unhold |
Collaborator
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: inbharajmani, raelga The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This was referenced Jul 3, 2026
openshift-merge-bot Bot
pushed a commit
that referenced
this pull request
Jul 6, 2026
Advances every component in config/config.yaml to its latest digest **except clusters-service**, which is intentionally pinned at the last-known-good digest `sha256:6a49b32…` (vcs-ref `18b5a25`, 2026-06-19). ## Why The automated bulk bump #5789 started failing `ci/prow/e2e-parallel` after the clusters-service image moved to `sha256:b17f6fe…` (vcs-ref `ee741db`). Bisecting the bump into per-component PRs isolated the culprit: - #5910 (hypershift only) — e2e-parallel green - #5911 (ACM/MCE only) — e2e-parallel green (merged) - #5920 (**clusters-service only**) — e2e-parallel **fails reproducibly** (2 consecutive runs) on the exact same specs: `test/e2e/complete_cluster_create_multiversion.go:172` "verify simple web app runs" for the **candidate 4.22 and 5.0** channels, with `route was never reachable: dial tcp 10.0.0.5:443: i/o timeout` — the control plane + node pools provision, but the data-plane ingress on the newest OCP channels never becomes reachable. 4.20/4.21 are unaffected. - #5912 (everything except CS) — only flaky/environmental failures (1 then 16, non-reproducible signature), consistent with the shared-CI ARM-throttling episode also hitting unrelated PRs (e.g. #5915, which edits only alert YAML). An image-digest bump is a behavioral change (the cluster runs the new build), and the CS-only PR reproduces a version-specific data-plane regression, so we pin CS at the good digest while letting the remaining components advance. Follow-up on the bad CS image is tracked in the Jira below. Jira: https://redhat.atlassian.net/browse/AROSLSRE-1395 ## Components bumped (clusters-service pinned, not bumped) | Component | Old | New | | --- | --- | --- | | acrpull | v0.1.23 | v0.1.24 | | arobit forwarder | v5.0.4 (06-19) | v5.0.4 (07-03) | | mdsd | 1.42.0-20260615 | 1.42.0-20260629 | | kube-events | 20260621.1 | 20260701.1 | | maestro (provider) | v1.8.2 (06-11) | v1.8.2 (06-26) | | hypershift | 9aeb1f3 | 488ef0e | | OADP velero-server | 1.6.1 (06-25) | 1.6.1 (07-02) | | OADP velero azure-plugin | 1.6.1 (06-25) | 1.6.1 (07-02) | | OADP velero hypershift-plugin | 1.6.1 (06-25) | 1.6.1 (07-02) | | kube-state-metrics | v2.19.0 (06-12) | v2.19.0 (06-30) | | maestro-agent-sidecar (nginx) | azl3.0.20260602 | azl3.0.20260616 | | image-sync/oc-mirror | 690892d | 5bfc996 | | **clusters-service** | **6a49b32 (pinned — good)** | **— (b17f6fe held back)** |
tmstff
pushed a commit
to tmstff/ARO-HCP
that referenced
this pull request
Jul 7, 2026
Advances every component in config/config.yaml to its latest digest **except clusters-service**, which is intentionally pinned at the last-known-good digest `sha256:6a49b32…` (vcs-ref `18b5a25`, 2026-06-19). ## Why The automated bulk bump Azure#5789 started failing `ci/prow/e2e-parallel` after the clusters-service image moved to `sha256:b17f6fe…` (vcs-ref `ee741db`). Bisecting the bump into per-component PRs isolated the culprit: - Azure#5910 (hypershift only) — e2e-parallel green - Azure#5911 (ACM/MCE only) — e2e-parallel green (merged) - Azure#5920 (**clusters-service only**) — e2e-parallel **fails reproducibly** (2 consecutive runs) on the exact same specs: `test/e2e/complete_cluster_create_multiversion.go:172` "verify simple web app runs" for the **candidate 4.22 and 5.0** channels, with `route was never reachable: dial tcp 10.0.0.5:443: i/o timeout` — the control plane + node pools provision, but the data-plane ingress on the newest OCP channels never becomes reachable. 4.20/4.21 are unaffected. - Azure#5912 (everything except CS) — only flaky/environmental failures (1 then 16, non-reproducible signature), consistent with the shared-CI ARM-throttling episode also hitting unrelated PRs (e.g. Azure#5915, which edits only alert YAML). An image-digest bump is a behavioral change (the cluster runs the new build), and the CS-only PR reproduces a version-specific data-plane regression, so we pin CS at the good digest while letting the remaining components advance. Follow-up on the bad CS image is tracked in the Jira below. Jira: https://redhat.atlassian.net/browse/AROSLSRE-1395 ## Components bumped (clusters-service pinned, not bumped) | Component | Old | New | | --- | --- | --- | | acrpull | v0.1.23 | v0.1.24 | | arobit forwarder | v5.0.4 (06-19) | v5.0.4 (07-03) | | mdsd | 1.42.0-20260615 | 1.42.0-20260629 | | kube-events | 20260621.1 | 20260701.1 | | maestro (provider) | v1.8.2 (06-11) | v1.8.2 (06-26) | | hypershift | 9aeb1f3 | 488ef0e | | OADP velero-server | 1.6.1 (06-25) | 1.6.1 (07-02) | | OADP velero azure-plugin | 1.6.1 (06-25) | 1.6.1 (07-02) | | OADP velero hypershift-plugin | 1.6.1 (06-25) | 1.6.1 (07-02) | | kube-state-metrics | v2.19.0 (06-12) | v2.19.0 (06-30) | | maestro-agent-sidecar (nginx) | azl3.0.20260602 | azl3.0.20260616 | | image-sync/oc-mirror | 690892d | 5bfc996 | | **clusters-service** | **6a49b32 (pinned — good)** | **— (b17f6fe held back)** |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bisecting the e2e-parallel regression on the automated image-digest PR #5789, which flipped from green to red at commit
128feb1.What
Bumps only the ACM operator bundle and MCE operator bundle digests (plus the regenerated
acm/helm charts and fixtures) to the latest values from #5789, on top of currentmain. Hypershift and all other components stay at theirmaindigests.Why
At the pass→fail boundary of #5789, ACM/MCE bundles were among the 6 changed digests. The failure is
CreateHCPClustertiming out across ~16 tests. This PR isolates the ACM/MCE bump so e2e-parallel tells us whether it alone breaks provisioning. Theacm/chart files were reused from #5789 head (no skopeo pull) and confirmed byte-identical, thenmake -C config/ materializeregenerated rendered config + fixtures.Testing
Relies on
ci/prow/e2e-parallel.make -C config/ detect-changeis clean. Diagnostic bisect PR — not intended to merge as-is.Special notes for your reviewer
Do not merge. Diagnostic bisect of #5789. Companion PRs isolate hypershift and the remaining components.